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New Experimental Designs for Paired Observations 

W. J. Youden and William S. Connor 

There are many experimental situations in which observations naturally occur m pairs. 
From v quantities to be compared, v(v— 1)/2 distinct pairs can be formed, so that for even 
moderately large v, the accompanying number of observations may be prohibitively large. 
Or, even if the work can be done, the precision of the estimates of the effects of the quantil ies 
and the experimental error may be greater than is needed. In either event, a subset of the 
pairs should be used. If the arithmetic is to be manageable, this subset must be chosen 
with care. Such a subset is described and its analysis is given in detail. 



1. Introduction 

In many experimental situations only two quan- 
tities can be observed at a time under comparable 
conditions, so that observations on a set of quantities 
naturally divide into pairs. If each quantity is 
compared with every other quantity of the set, the 
statistical analysis of the observations is simple. 
However, if the number of quantities is large, the 
amount of work required to make observations on 
all of the pairs may be prohibitive. Even if the 
work can be done, the results of the experiment may 
be more precise than is needed. In either event, it 
is desiiable to obesrve only part of the pairs. The 
selection of this subset of pairs must be made with 
care if the statistical analysis is to remain relatively 
simple. 

An example will help to clarify the problem. 
Suppose that 36 quantities are to be studied, two at 
a time. From them there are 630 distinct pairs 
that can be formed. In this paper consideration 
will be given to particularly attractive subsets 
called two-group arrangements, which require from 
324 down to 35 of these pairs. 

To illustrate the statistical analysis of such an 
arrangement, a typical subset for the case of eight 
quantities is analyzed, using data on thermometer 
calibrations. 

2. The Two-Group Arrangement 

It is common scientific practice to compare new 
objects under study with one or more standards. 
An important function of the National Bureau of 
Standards is the calibration of thermometers, meter 
bars, and other devices for industrial and other uses. 
These calibrations are made by comparing the new 
objects with established standards. 

Perhaps the most usual situation is that in which 
there is a single standard, which may be designated 
by S. If there are six new objects to be calibrated, 
then a common practice is to pair each new object 
with S. Thus, denoting the new objects by num- 
bers, the pairs are as follows: 



SI 



S3 S5 



S2 S4 



S6 



This traditional experimental procedure suggests 
the two-group arrangement, which consists of divid- 
ing the v objects under study into two groups of m 
and n objects, respectively, (v=m+ri), and of pairing 
every object from one group with every object from 
the other group. 1 No other pairs are formed. 

In the situation just considered the standard is 
the only object in one group so that m = l and n=6. 
This arrangement provides information about the 
standard, the new objects, and the pairs, but gives 
no information about the experimental error. To 
obtain such information it is natural to run each pair 
again, so that there are 24 observations altogether. 

Among these observations the standard occurs 12 
times and the new objects tw r ice each. This lays 
heavy emphasis on comparisons between the new 
objects and the standard but less emphasis on com- 
parisons among the new objects. Thus if a is the 
true standard deviation of an observation, then the 
standard deviation of a comparison of the first kind 
is o- and of the second kind is 1.4o\ 

Another two-group arrangement results from as- 
signing the standard and new objects 1 and 2 to 
one group and new objects 3, 4, 5, and 6 to the 
other. In this case ?n = 3, n=4. and the pairs are 
the following. 

S3 S4 S5 S6 



13 



23 



14 



24 



15 



25 



16 



26 



There are 24 observations and as much information 
about the experimental error as in the preceding 
arrangement. The standard is put on the same 
footing as the new objects because it is observed 4 
times and the new objects either 3 or 4 times. This 
is reflected among the comparisons, for the standard 
deviation of the comparison between any two objects 
in the first group is a, in the second group is 1.15c-, 
and between an object in the first group and one in 
the second is a. Thus there is no loss in the preci- 
sion of comparisons that involve the standard, and 
there is a substantial gain in the precision of other 
comparisons. 

i When m =n, the arrangement has been called Group Divisible. See, R. C. 
Bose and T. Shimamoto, Classification and analysis of partially balanced 
incomplete block designs with two associate classes, J. Am. Statistical Assoc. 47 
151 (1952). 
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The problem posed in the introduction can now 
be resolved in many ways. If the 36 quantities are 
divided into two groups of 18 each, 324 pairs will 
be formed. At the other extreme is the division 
into 1 and 35, which results in only 35 pairs. 

3. Application to Thermometer Calibration 

The authors asked the Thermometry Section of 
the National Bureau of Standards to intercom pare 
eight thermometers, using the two-group arrange- 
ment. The usual practice of the section is to read 
the thermometers in sequence in a bath with slowly 
rising temperature and then to read them in reverse 
order. This device effectively compensates for 
changes in the bath temperature, provided that the 
temperature changes at a constant rate. The 
effectiveness of the two-group arrangement, however, 
does not depend on a constant rate of change in 
temperature. 

The thermometers were partly im.mersed in a bath 
of distilled water, and were read through a telescope 
mounted a short distance away. The temperature 
of the bath was at approximately 40° C at the start 
of the readings, but rose gradually throughout the 
experiment. There were short pauses of irregular 
length between pairs of readings. 

The eight thermometers were divided into 2 
groups of 4 each, containing thermometers 1, 2, 3, 
and 4 and 5, 6, 7, and 8, respectively. The read- 
ings are given in table 1 in the order in which they 
were obtained. 2 

The computations can be simplified by subtract- 
ing some convenient number from each observation. 
Accordingly, all subsequent calculations are based 
on the observations in table 1 after subtracting 40 
from each of them. 

2 The thermometers were randomized within the pairs and the pairs within 
the runs. 



The mathematical model underlying the statistical 
analysis is based on the following considerations. 
Let M be a reference temperature in the range of 
temperatures of the bath during the experiment. 
At the time of measurement of the jth pair of ther- 
mometers, the temperature of the bath will be 
M+pj, where pj is defined by this condition. 

Next, suppose that the ith thermometer belongs 
to the jth pair, and let x tJ denote the observed tem- 
perature for this thermometer when the jth pair is 
read. Then the difference between the observed 
temperature x ti and the true bath temperature M-^-pj 
will consist of two parts: a systematic error t h pe- 
culiar to the ith thermometer, and a random reading 
error e ih i. e., 



x {J — (M+p j )=t i +e ij 

x ij =M+t i +p j +e ij . 



or 



By imposing the restrictions ^ J t i =^2p j =0, M is 

uniquely defined. 

The constants M, t t , and p 3 and the error e^ are 
unknown but can be estimated from the data. It 
is assumed that the errors associated with different 
readings are independent and come from the same 
population of errors. This population is assumed to 
have mean zero and standard deviation o~, which 
may or may not be known. 

The following calculations will show how to esti- 
mate the constants and the standard deviation. 3 
Estimates of the t J s are of especial interest, since 
they may be used to calibrate a new thermometer in 
terms of a standard. Estimates are denoted by 

3 Derivations of formulas are given in the appendix. 



Table 1. Temperature readings in order of time 



Run 


1 
1 


2 


3 


4 


Pair 


Ther- 
mometer 


Reading 


Pair 


Ther- 
mometer 


Reading 


Pair 


Ther- 
mometer 


Reading 


Pair 


Ther- 
mometer 


Reading 


1 

2 

3____ 
4 


{ ! 
{ I 
{ 1 
I I 


°C 
40.00 

39. 99 

40. 08 
40. 13 

40. 15 
40. 17 

40. 13 
40. 05 


5 

6 

7 

8 


{ i 
{ i 

i i 

{ s 


°C 
40. 18 
40. 18 

40. 07 
40. 19 

40. 10 
40. 18 

40. 17 
40. 13 


9 

10----- 

11 

12 


{ ! 

{ 5 
{ I 
{ f 


°C 
40. 23 
40. 22 

40. 24 
40. 15 

40. 12 
40. 20 

40. 23 
40. 16 


13 

14 

15 

16 


{ i 
i i 

l I 

{ I 


°C 
40. 26 
40. 28 

40. 15 
40. 20 

40. 27 
40. 30 

40. 21 
40.31 
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carets. For example, t t is the estimate of ti. 

To analyze the coded data it is convenient to 
compute an auxiliary quantity, D, for each thermom- 
eter. Thus D u the D for the ?th thermometer, is 
computed as follows. For each pair that contains 
the v"th thermometer the difference between the 
reading for the ith thermometer and the reading for 
the other thermometer of the pair is computed. 
The sum of these differences is D t . For example, 



A=[0-(-.01)] + (.10-.18)- 



-(.16-.23) 

+ (.21-.31) = 



.24. 



Let the group that contains m thermometers be 
called group 1 , and the group that contains n ther- 
mometers be called group 2. Let the sum of the 
D's for the thermometers in group 1 be denoted by 
Si, and in group 2 by S 2 . Then the D's may be used 
to estimate the correction for the ith thermometer 
by the following formulas: 4 

t i =(vD i —S l )/vn 
if i is in group 1 , and 



* It sometimes happens that the temperatures or other quantities are not 
observed directly, but instead the diiTerences between the quantities in the pairs 
are recorded. Although in this case M and the p's cannot be estimated, the 
>'s still are estimable by these formulas. 



t^ivDt-SJ/rm 

if iis in group 2. For example, for the first thermom- 
eter 



tf 1 =(8A-S 1 )/32=(-1.92 + .07)/32= -.05781. 

If a is unknown from past experience, it may be 
calculated from the data. This calculation is made 
quite simply by working with the differences between 
the readings within a pair. Let the difference 
without regard to sign for the jth pair be designated 
by dj. Then a is estimated from the formula 5 



2(mn-m-n+l)$ 2 =Jld*-Jlt i D i . 

The computations may be systematized by use of 
table 2, in which the estimates of the £'s and y^U ? D, 

2 = 1 

are found. 



5 When just the diiTerences are observed, it is convenient to do the analysis in 
terms of the standard deviation of the differences, which may conveniently be 
denoted by <r<t. This formula and others below apply in this case, too, provided 
a is replaced by o- d /2. 



Table 2. Calculation of the thermometer effects 







Group 2 (/?) 
thermometer 


Calculations 




5 


6 


7 


8 


E 


D 


SO 


A 

32/ 


t 


A 

Dt 


w <D 

- £ 

!§ 
11 


1 


.16 
.23 


.10 

.12 




-.01 


.21 
.31 


.71 


-.24 


-1.92 


-I.85 


-.05781 


.01388 


2 


.30 
.27 


.23 
.22 


.19 
.07 


.17 
.15 


.^9 
.71 


.18 


1.44 


1.51 


.04719 


.00849 


3 


.13 

.08 


.28 
.26 


.20 
.12 


.18 
.18 


:B 


.15 


1.20 


1.27 


.03969 


.00595 


4 


.13 
.17 


.05 
.13 


.20 
.15 


.24 


•53 

.69 


-.16 


-1.28 


-1.21 


-.03731 


.00605 


c 
o 

o 

o 
o 

o 


E 


.75 .72 


.79 .66 


.33 .59 


.88 .71 














D 


.03 


.13 


-.26 


.17 




-.07 
.07 










QD 


.21+ 


i.o4- 


-2.08 


1.36 






-.56 

.56 








A 

32 * 


.17 


.97 


-2.15 


1.29 








-.28 
.28 






t 


.00531 


.03031 


-.06719 


.04031 










-.00874 
.00874 




ot 


.00016 


.00394 


.017^7 


.00685 












.01437 
.02842 
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The coded readings are entered in the upper 
left-hand part of the table, where every cell corre- 
sponds to some pair. For example, the first pair is 
put into the cell in row 1 and column 7, with the 
reading for thermometer 1 recorded in the upper 
right-hand corner and for thermometer 7 in the 
lower left-hand corner. By so recording the read- 
ings, each row and column is divided into subrows 
a ad sub columns. 

The remaining rows and columns are for calcula- 
tions, which it is believed are self-evident. In 

A A 

general, row SD is replaced by vD and 32 1 by (vm) t. 

Likewise, column SD is replaced by vD and 32 ^ by 

(m)t. Several checks are available: (1) the sum of 
the entries in row 2 must equal the sum of the 
entries in column 2, and (2) the sums of the other 
corresponding rows and columns, except the last, 
must be of different sign but of the same absolute 
value. In the table these quantities appear along 
the diagonal. 

The standard deviation is estimated from the 
formula given above. The differences d may easily 

V 

be calculated from table 1, and 2 UD t from 

table 2. The differences and calculations on them 
are given in table 3. 

Table 3. Calculation of the standard deviation 



Pair (J) 


di 


d) 


Pair (j) 


di 


A 


1 


0.01 
.05 
.02 
.08 
.00 
.12 
.08 
.04 


0. 0001 
.0025 
.0004 
.0064 
.0000 
.0144 
. 0064 
.0016 


9 


0.01 
.09 
.08 
.07 
.02 
.05 
.03 
.10 


0. 0001 
.0081 
.0064 
.0049 
.0004 
.0025 
.0009 
.0100 


2 


10 


3 


11 

12 

13 


4 


5 


6 . 


14 


7 


15 


8 . 


16 






16 o 


8 A 

S UDi = 


03437+.02842 = .0628, 

18^2 =.0651-. 


0628 =.0023 


, £=.0114. 



Two thermometers can be compared by finding 
the difference between their estimated effects. To 
judge the significance of such a difference, it is de- 
sirable to know the standard deviation of the differ- 
ence. If i and V both are in group 1, then the 
square of the standard deviation of the difference is 

o-, 2 a =4:a 2 /n; 

if both are in group 2, then 

<t? A =4:o- 2 /m; 
n-w 

and if i is in group 1 but %' is in group 2, then 
a ? * =2(v— \)a 2 lmni 



If o- 2 is not known, then its estimate is used. 

As an example, consider thermometers 1 andv2. 
The appropriate formula is the first one above, "so 
that 



a* 2 a =.0114. 



Just as it has been possible to intercompare the 
thermometers even though in some cases a particular 
pair of thermometers were never at the same tem- 
perature, so also it is possible to determine the rela- 
tive temperatures of the bath when each of the ?nn 
pairs of thermometers were read even though the 
temperatures were read with different thermometers 
with unknown corrections. It may sometimes be 
important to ascertain the character of the drift or 
changes taking place in the experimental system. 
In the example given, matters were arranged so that 
there was an approximately linear drift upward in 
the bath temperature. Table 4 reflects this condi- 
tion, the values being computed as is indicated 
below. 

Tabee 4. Average temperatures of the pairs referred to 40° C 



Pair 


Uncor- 
rected 
average 


Cor- 
rected 
average 


Pair 


Uncor- 
rected 
average 


Cor- 
rected 
average 


1 

2 


°C 
-0. 005 
.105 
.160 
.090 
.180 
.130 
.140 
.150 


°C 

0.058 
.082 
.116 
.094 
.140 
.140 
.154 
.166 


9 


°C 
0.225 
.195 
.160 
.195 
.270 
.175 
.285 
.260 


°C 

0.186 
.194 
.174 
.221 
.235 
.22S 
.259 
.269 


10 

11 

12 

13 


3 


4 


5 

6 


14 


7 


15... 


8 


16 







The averages after correction for thermometers 
exhibit the upward trend much more clearly than do 
the crude, uncorrected averages. 

The uncorrected averages for the jth pair is 
simply the arithmetic average of the two readings in 
the pair. The corrected average is the uncorrected 
average adjusted for the systematic errors of the 
thermometers that occur in the jth pair. In symbols 

it is M+Pj. 

The estimate of M is 

A *' ran m ^ 

2mnM=J2 S Xij+(m—n) XI U, 

i=l j=l i = l 

which, in the case at hand, reduces to 

A / 8 16 \ 

M=(S2>„)/32, 

\i=l i=l / 

A 

the grand mean of the readings. Thus M= 5.43/32 
= .16969. These formulas should be used with the 
understanding that Xy=0 if the ith thermometer 
does not occur in the jth pair. 
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Thus far all values have been given in coded form 
and the adjusted thermometer readings in terms of 
systematic deviations from the reference temperature 
M. It may be of interest to estimate readings for 
all thermometers at temperature M. These decoded 
estimated readings, calculated by the formula 

A A 

J/+40 + **, are as follows: 



Thermom- 
eter 


Temper- 
ature 


Thermom- 
eter 


Temper- 
ature 


1 
2 
3 
4 


° C. 
40.11 
40.22 
40.21 
40. 13 


5 
6 

7 
8 


°C. 

40.18 
40.20 
40.10 
40.21 



The estimate of pj is obtained by a simple adjust- 
ment of the observations in the jth pair. If i and i' 
are the thermometers in the jth pair, then 

a - A , A A 

2pj=Xij+Xi>j—2M— U— U'. 

For example, fory=2, i=5, i'=3, and 

2p 2 = .08 + .13-2(.16969)-.00531-.0396y, 

so that p 2 = — .08719. 

It now is possible to exhibit the decomposition of 
x 52 into its parts. Thus 

A A A A 

x 52 =M+t 5 +p 2 +e 52 

.08 = .16969 + .00531 + (-.08719) + (-.00781). 

It is interesting to note that the estimated error in 
this particular reading is of about the same magni- 
tude as <r. 

The fundamental importance of the arrangement 
is that it makes it possible to intercompare the 
thermometers and to limit the error arising from 
fluctuations in the bath temperature to those tem- 
perature changes that take place in the very short 
interval required to read two thermometers. Tem- 
perature changes from one pair to another do not 
contribute to the error of measurement. This 
technique is applicable in all cases where either the 
apparatus or the environment may drift or undergo 
unpredictable changes. 

4. Appendix 

4.1. Derivation of Estimates 

Let the group that contains m objects be denoted 
by G\ y and the group that contains n objects be 
denoted by G 2 . Then the reduced normal equations 



for estimating the treatment (thermometer) effects 
are 



for i in Gi and 



a * A 

u=m-\-l 



A m a 

mt i —2_ l t u =D i 



(1) 



(2) 



for i in G 2 . 

Summing over the treatments in G l} eq (1) becomes 



971 a » a m 

nJ2t u —m J2 t u =^D u . 

U=l tt=7W-f-l U = l 



Then imposing the restriction 

2X=o, 



it is clear that 



771 A 771 

(n+m)St,=S7J,. 

M=l U = l 



(3) 



(4) 



(5) 



Similarly, using eq (2), summing over the treatments 
in G 2) and applying eq (4), obtain 



(n+m) 2 t u = S Du- 

u=m-\-\ u=m+l 



(6) 



From eq (1) and (6) it follows for i in G x that 

a v 

n(n+m)ti=(n+m)Di+ XI D u 

M = ra-{-l 



or since X D u =0, 



vnti=vDi—y~*,D„. 



Similarly, for i in G 2 , 



(7) 



(8) 



vmt i =vD i — XI D u . 

W = 77l4-1 

4.2 Derivation of Variance 



For random variables x and y let V(x) and Cov(x,y) 
denote, respectively, the variance of x and the 
co variance of x and y. Then for i and i r in Q X) 



V(D i )=2ma 2 , Cov{D i} D it ) = Q. 
From eq (7) and (9), 

Similarly, for i and i' both in G 2 , 



(9) 



(10) 
(11) 
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For i in G x and %' in G 2 it is convenient to use the 
formula 



V(t if - A t if ) = (C u +Cr 



-2C w y, 



(12) 



where C rs is the element in the rth row and sth 
column of the inverse of the coefficient matrix of 
the reduced normal equations. From eq (7) and (8), 

C u =2(v—l)/vn, C i > i >=2(v—l)!vm, C^ = 0. 

Hence eq (12) becomes 

V(t i -%)=2(v-l)^!mn. (13) 



4.3 Derivation of Estimate of a 

The differences dj form a basis for the space , 
which consists of the error space and the space of 

the t's. Therefore, the sum of squares due to the 
d's can be partitioned into two orthogonal parts, 
one due to error and one due to treatments. Since 

the sum of squares due to treatments is ( 2 ^iPi ) 2, 

twice the sum of squares due to error is 

A mn v a 

2(mn,-m-7i+l)ff 2 =S $— S UD t . 
Washington, September 25, 1954. 
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